As an important variant of entity alignment (EA), multi-modal entity alignment (MMEA) aims to discover identical entities across different knowledge graphs (KGs) with multiple modalities like images. However, current MMEA algorithms all adopt KG-level modality fusion strategies but ignore modality differences among individual entities, hurting the robustness to potential noise involved in modalities (e.g., unidentifiable images and relations). In this paper we present MEAformer, a multi-modal entity alignment transformer approach for meta modality hybrid, to dynamically predict the mutual correlation coefficients among modalities for instance-level feature fusion. A modal-aware hard entity replay strategy is also proposed for addressing vague entity details. Extensive experimental results show that our model not only achieves SOTA performance on multiple training scenarios including supervised, unsupervised, iterative, and low resource, but also has limited parameters, optimistic speed, and good interpretability. Our code will be available soon.

In recent years, the number of parameters of one deep learning (DL) model has been growing much faster than the growth of GPU memory space. People who are inaccessible to a large number of GPUs resort to heterogeneous training systems for storing model parameters in CPU memory. Existing heterogeneous systems are based on parallelization plans in the scope of the whole model. They apply a consistent parallel training method for all the operators in the computation. Therefore, engineers need to pay a huge effort to incorporate a new type of model parallelism and patch its compatibility with other parallelisms. For example, Mixture-of-Experts (MoE) is still incompatible with ZeRO-3 in Deepspeed. Also, current systems face efficiency problems on small scale, since they are designed and tuned for large-scale training. In this paper, we propose Elixir, a new parallel heterogeneous training system, which is designed for efficiency and flexibility. Elixir utilizes memory resources and computing resources of both GPU and CPU. For flexibility, Elixir generates parallelization plans in the granularity of operators. Any new type of model parallelism can be incorporated by assigning a parallel pattern to the operator. For efficiency, Elixir implements a hierarchical distributed memory management scheme to accelerate inter-GPU communications and CPU-GPU data transmissions. As a result, Elixir can train a 30B OPT model on an A100 with 40GB CUDA memory, meanwhile reaching 84% efficiency of Pytorch GPU training. With its super-linear scalability, the training efficiency becomes the same as Pytorch GPU training on multiple GPUs. Also, large MoE models can be trained 5.3x faster than dense models of the same size. Now Elixir is integrated into ColossalAI and is available on its main branch.

Neural network pruning has been a well-established compression technique to enable deep learning models on resource-constrained devices. The pruned model is usually specialized to meet specific hardware platforms and training tasks (defined as deployment scenarios). However, existing pruning approaches rely heavily on training data to trade off model size, efficiency, and accuracy, which becomes ineffective for federated learning (FL) over distributed and confidential datasets. Moreover, the memory- and compute-intensive pruning process of most existing approaches cannot be handled by most FL devices with resource limitations. In this paper, we develop FedTiny, a novel distributed pruning framework for FL, to obtain specialized tiny models for memory- and computing-constrained participating devices with confidential local data. To alleviate biased pruning due to unseen heterogeneous data over devices, FedTiny introduces an adaptive batch normalization (BN) selection module to adaptively obtain an initially pruned model to fit deployment scenarios. Besides, to further improve the initial pruning, FedTiny develops a lightweight progressive pruning module for local finer pruning under tight memory and computational budgets, where the pruning policy for each layer is gradually determined rather than evaluating the overall deep model structure. Extensive experimental results demonstrate the effectiveness of FedTiny, which outperforms state-of-the-art baseline approaches, especially when compressing deep models to extremely sparse tiny models.

Significant progress has been made in learning image classification neural networks under long-tail data distribution using robust training algorithms such as data re-sampling, re-weighting, and margin adjustment. Those methods, however, ignore the impact of data imbalance on feature normalization. The dominance of majority classes (head classes) in estimating statistics and affine parameters causes internal covariate shifts within less-frequent categories to be overlooked. To alleviate this challenge, we propose a compound batch normalization method based on a Gaussian mixture. It can model the feature space more comprehensively and reduce the dominance of head classes. In addition, a moving average-based expectation maximization (EM) algorithm is employed to estimate the statistical parameters of multiple Gaussian distributions. However, the EM algorithm is sensitive to initialization and can easily become stuck in local minima where the multiple Gaussian components continue to focus on majority classes. To tackle this issue, we developed a dual-path learning framework that employs class-aware split feature normalization to diversify the estimated Gaussian distributions, allowing the Gaussian components to fit with training samples of less-frequent classes more comprehensively. Extensive experiments on commonly used datasets demonstrated that the proposed method outperforms existing methods on long-tailed image classification.

Recent works have impressively demonstrated that there exists a subnetwork in randomly initialized convolutional neural networks (CNNs) that can match the performance of the fully trained dense networks at initialization, without any optimization of the weights of the network (i.e., untrained networks). However, the presence of such untrained subnetworks in graph neural networks (GNNs) still remains mysterious. In this paper we carry out the first-of-its-kind exploration of discovering matching untrained GNNs. With sparsity as the core tool, we can find \textit{untrained sparse subnetworks} at the initialization, that can match the performance of \textit{fully trained dense} GNNs. Besides this already encouraging finding of comparable performance, we show that the found untrained subnetworks can substantially mitigate the GNN over-smoothing problem, hence becoming a powerful tool to enable deeper GNNs without bells and whistles. We also observe that such sparse untrained subnetworks have appealing performance in out-of-distribution detection and robustness of input perturbations. We evaluate our method across widely-used GNN architectures on various popular datasets including the Open Graph Benchmark (OGB).

We launch EVA, a vision-centric foundation model to explore the limits of visual representation at scale using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVISv1.0 dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can greatly stabilize the training and outperform the training from scratch counterpart with much fewer samples and less compute, providing a new direction for scaling up and accelerating the costly training of multi-modal foundation models. To facilitate future research, we release all the code and models at https://github.com/baaivision/EVA.

强有力的对手例子是评估和增强深神经网络鲁棒性的关键。流行的对抗性攻击算法使用梯度上升最大化非cave损失函数。但是，每种攻击的性能通常对由于信息不足（仅一个输入示例，几乎没有白色盒子源模型和未知的防御策略）而敏感。因此，精心设计的对抗性示例容易过度拟合源模型，从而将其转移性限制在身份不明的架构上。在本文中，我们提出了多种渐近正态分布攻击（Multianda），这是一种新颖的方法，可以明确表征来自学习分布的对抗性扰动。具体而言，我们通过利用随机梯度上升（SGA）的渐近正态性能（SGA）的优势来近似于扰动，然后将整体策略应用于此过程，以估算高斯混合模型，以更好地探索潜在的优化空间。从学习分布中绘制扰动使我们能够为每个输入生成任何数量的对抗示例。近似后验实质上描述了SGA迭代的固定分布，该分布捕获了局部最佳距离周围的几何信息。因此，从分布中得出的样品可靠地保持转移性。我们提出的方法通过对七个正常训练和七个防御模型进行广泛的实验，超过了对具有或没有防御的深度学习模型的九个最先进的黑盒攻击。

行业分配根据预定义的行业分类系统（ICS）将公司分配给行业，这对于大量关键业务实践至关重要，从公司的运营和战略决策到政府机构的经济分析。三种专家知识对于有效行业分配至关重要：基于定义的知识（即每个行业的专家定义），基于结构的知识（即ICS中指定的行业之间的结构关系）和基于任务的知识（即，域专家执行的事先公司行业任务）。现有的行业分配方法仅利用基于任务的知识来学习将未分配的公司分类为行业的模型，并忽略基于定义和基于结构的知识。此外，这些方法仅考虑已分配了公司的哪个行业，但忽略了基于分配的知识的时间特异性，即在任务发生时。为了解决现有方法的局限性，我们提出了一种新颖的基于深度学习的方法，该方法不仅无缝整合了三种类型的行业分配知识，而且还考虑了基于分配的知识的特定时间。从方法上讲，我们的方法具有两种创新：动态行业表示和分层分配。前者通过通过我们提出的时间和空间聚集机制整合了三种类型的知识，将行业代表为一系列特定时间的向量。后者将行业和公司的表现作为投入，计算将公司分配给不同行业的可能性，并将公司分配给具有最高概率的行业。

尽管以前基于图的多视图聚类算法已经取得了重大进展，但其中大多数仍面临三个限制。首先，他们经常遭受高计算复杂性的困扰，这限制了他们在大规模场景中的应用。其次，他们通常在单视图级别或视图传感级别上执行图形学习，但经常忽略单视图和共识图的联合学习的可能性。第三，其中许多人依靠$ k $ - 表示光谱嵌入的离散化，这些嵌入缺乏直接使用离散群集结构直接学习图形的能力。鉴于此，本文通过统一和离散的两部分图（UDBGL）提出了一种有效的多视图聚类方法。具体而言，基于锚的子空间学习被合并为从多个视图中学习特定的二分化图，并利用双方图融合来学习具有自适应重量学习的视图 - 谐镜双分歧图。此外，施加Laplacian等级约束以确保融合的两分图具有离散的群集结构（具有特定数量的连接组件）。通过同时制定特定视图的两分图学习，视图 - 共表的两分图学习以及离散的群集结构学习到统一的目标函数中，然后设计有效的最小化算法来解决此优化问题，并直接实现离散的聚类解决方案解决方案解决方案解决方案解决方案。不需要其他分区，这特别是数据大小的线性时间复杂性。各种多视图数据集的实验证明了我们的UDBGL方法的鲁棒性和效率。

多用户延迟约束调度在许多现实世界应用中都很重要，包括无线通信，实时流和云计算。然而，它提出了一个关键的挑战，因为调度程序需要做出实时决策，以确保没有系统动力学的先前信息，这可能是时间变化且难以估算的。此外，许多实际情况都遭受了部分可观察性问题的影响，例如，由于感应噪声或隐藏的相关性。为了应对这些挑战，我们提出了一种深入的强化学习（DRL）算法，称为Recurrent Softmax延迟深层双重确定性策略梯度（$ \ Mathtt {RSD4} $），这是一种基于数据驱动的方法，基于部分观察到的Markov决策过程（POMDP）配方。 $ \ mathtt {rsd4} $分别通过拉格朗日双重和延迟敏感的队列保证资源和延迟约束。它还可以通过复发性神经网络（RNN）启用的记忆机制有效地解决部分可观察性，并引入用户级分解和节点级别的合并以确保可扩展性。对模拟/现实世界数据集的广泛实验表明，$ \ mathtt {rsd4} $对系统动力学和部分可观察到的环境是可靠的，并且在现有的DRL和非基于DRL的方法上实现了卓越的性能。